This invention relates generally to information delivery systems, and more particularly to delivery systems that adapt information encoded as object-based compressed bitstreams to available bit rates of a network.
Recently, a number of standards have been developed for communicating encoded information. For video sequences, the most widely used standards include MPEG-1 (for storage and retrieval of moving pictures), MPEG-2 (for digital television) and H.263, see ISO/IEC JTC1 CD 11172, MPEG, xe2x80x9cInformation Technologyxe2x80x94Coding of Moving Pictures and Associated Audio for Digital Storage Media up to about 1.5 Mbit/sxe2x80x94Part 2: Coding of Moving Pictures Information,xe2x80x9d 1991, LeGall, xe2x80x9cMPEG: A Video Compression Standard for Multimedia Applications,xe2x80x9d Communications of the ACM, Vol. 34, No. 4, pp. 46-58, 1991, ISO/IEC DIS 13818-2, MPEG-2, xe2x80x9cInformation Technologyxe2x80x94Generic Coding of Moving Pictures and Associated Audio Informationxe2x80x94Part 2: Video,xe2x80x9d 1994, ITU-T SG XV, DRAFT H.263, xe2x80x9cVideo Coding for Low Bitrate Communication,xe2x80x9d 1996, ITU-T SG XVI, DRAFT13 H.263+Q15-A-60 rev.0, xe2x80x9cVideo Coding for Low Bitrate Communication,xe2x80x9d 1997.
These standards are relatively low-level specifications that primarily deal with the spatial and temporal compression of video sequences. As a common feature, these standards perform compression on a per frame basis. With these standards, one can achieve high compression ratios for a wide range of applications.
Newer video coding standards, such as MPEG-4 (for multimedia applications), see xe2x80x9cInformation Technologyxe2x80x94Generic coding of audio/visual objects,xe2x80x9d ISO/IEC FDIS 14496-2 (MPEG4 Visual), Nov. 1998, allow arbitrary-shaped objects to be encoded and decoded as separate video object planes (VOP). The objects can be visual, audio, natural, synthetic, primitive, compound, or combinations thereof. Video objects are composed to form compound objects or xe2x80x9cscenes.xe2x80x9d
The emerging MPEG-4 standard is intended to enable multimedia applications, such as interactive video, where natural and synthetic materials are integrated, and where access is universal. MPEG-4 allows for content based interactivity. For example, one might want to xe2x80x9ccut-and-pastexe2x80x9d a moving figure or object from one video to another. In this type of application, it is assumed that the objects in the multimedia content have been identified through some type of segmentation process, see for example, U.S. Pat. No. 6,400,846xe2x80x9cMethod for Ordering Image Spaces to Search for Object Surfacesxe2x80x9d filed on Jun. 4, 1999 by Lin et al.
In the context of video transmission, these compression standards are needed to reduce the amount of bandwidth (available bit rate) that is required by the network. The network may represent a wireless channel or the Internet. In any case, the network has limited capacity and a contention for its resources must be resolved when the content needs to be transmitted.
Over the years, a great deal of effort has been placed on architectures and processes that enable devices to transmit the content robustly and to adapt the quality of the content to the available network resources. When the content has already been encoded, it is sometimes necessary to further convert the already compressed bitstream before the stream is transmitted through the network to accommodate, for example, a reduction in the available bit rate.
Bit stream conversion or xe2x80x9ctranscodingxe2x80x9d can be classified as bit rate conversion, resolution conversion, and syntax conversion. Bit rate conversion includes bit rate scaling and conversion between a constant bit rate (CBR) and a variable bit rate (VBR). The basic function of bit rate scaling is to accept an input bitstream and produce a scaled output bitstream which meets new load constraints of a receiver. A bit stream scaler is a transcoder, or filter, that provides a match between a source bitstream and the receiving load.
As shown in FIG. 1, typically, scaling can be accomplished by a transcoder 100. In a brute force case, the transcoder includes a decoder 110 and encoder 120. A compressed input bitstream 101 is fully decoded at an input rate Rin, then encoded at a new output rate Rout 102 to produce the output bitstream 103. Usually, the output rate is lower than the input rate. However, in practice, full decoding and full encoding in a transcoder is not done due to the high complexity of encoding the decoded bitstream.
Earlier work on MPEG-2 transcoding has been published by Sun et al., in xe2x80x9cArchitectures for MPEG compressed bitstream scaling,xe2x80x9d IEEE Transactions on Circuits and Systems for Video Technology, Apr. 1996. There, four methods of rate reduction, with varying complexity and architecture, were presented.
FIG. 2 shows an example method. In this architecture, the video bitstream is only partially decoded. More specifically, macroblocks of the input bitstream 201 are variable-length decoded (VLD) 210. The input bitstream is also delayed 220 and inverse quantized (IQ) 230 to yield discrete cosine transform (DCT) coefficients. Given the desired output bit rate, the partially decoded data are analyzed 240 and a new set of quantizers is applied at 250 to the DCT blocks. These re-quantized blocks are then variable-length coded (VLC) 260 and a new output bitstream 203 at a lower rate can be formed. This scheme is much simpler than the scheme shown in FIG. 1 because the motion vectors are re-used and an inverse DCT operation is not needed.
More recent work by Assuncao et al., in xe2x80x9cA frequency domain video transcoder for dynamic bit-rate reduction of MPEG-2 bitstreams,xe2x80x9d IEEE Transactions on Circuits and Systems for Video Technology, pp. 953-957, December 1998, describe a simplified architecture for the same task. They use a motion compensation (MC) loop, operating in the frequency domain for drift compensation. Approximate matrices are derived for fast computation of the MC blocks in the frequency domain. A Lagrangian optimization is used to calculate the best quantizer scales for transcoding.
Other work by Sorial et al, xe2x80x9cJoint transcoding of multiple MPEG video bitstreams,xe2x80x9d Proceedings of the International Symposium on Circuits and Systems, May 1999, presents a method of jointly transcoding multiple MPEG-2 bitstreams, see also U.S. patent application Ser. No. 09/410,552xe2x80x9cEstimating Rate-Distortion Characteristics of Binary Shape Data,xe2x80x9d filed Oct. 1, 1999 by Vetro et al.
According to prior art compression standards, the number of bits allocated for encoding texture information is controlled by a quantization parameter (QP). The above papers are similar in that changing the QP based on information that is contained in the original bitstream reduces the rate of texture bits. For an efficient implementation, the information is usually extracted directly in the compressed domain and may include measures that relate to the motion of macroblocks or residual energy of DCT blocks. This type of analysis can be found in the bit allocation analyzer.
Although in some cases, the bitstream can be preprocessed, it is still important that the transcoder operates in real-time. Therefore, significant processing delays on the bitstream cannot be tolerated. For example, it would not be feasible for the transcoder to extract information from a group of frames, then transcode the content based on this look-ahead information. This would not work for live broadcasts, or video conferencing. Although it is possible to achieve better transcoding results in terms of quality due to better bit allocation, such an implementation for real-time applications is impractical.
It is also important to note that classical methods of transcoding are limited in their ability to reduce the bit rate. In other words, if only the QP of the outgoing video is changed, then there is a limit to how much one may reduce the rate. The limitation in reduction is dependent on the bitstream under consideration. Changing the QP to a maximum value will usually degrade the content of the bitstream significantly. Another alternative to reducing the spatial quality is to reduce the temporal quality, i.e., drop or skip frames. Again, skipping too many frames will also degrade the quality significantly. If both reductions are considered, then the transcoder is faced with a trade-off in spatial versus temporal quality.
This concept of such a spatio-temporal trade-off can also be considered in the encoder. However, not all video coding standards support frame skipping. For example, in MPEG-1 and MPEG-2, the Group of Picture (GOP) structure is pre-determined, i.e., the Intra frame period and distance between anchor frames is fixed. As a result, all pictures must be encoded. To get around this temporal constraint, the syntax does allow macroblocks to be skipped. If all macroblocks in a frame are skipped, then the frame has essentially been skipped. At least one bit is used for each macroblock in the frame to indicate this skipping. This can be inefficient for some bit rates.
The H.263 and MPEG-4 standards do allow frame skipping. Both standards support a syntax that allows a reference to be specified. However, their frame skipping has mainly been used to satisfy buffer constraints. In other words, if the buffer occupancy is too high and in danger of overflow, then the encoder will skip a frame to reduce the flow of bits into the buffer and give the buffer some time to send its current bits.
A more sophisticated use of this syntax allows one to make the spatio-temporal trade-offs in non-emergency situations, i.e., code more frames at a lower spatial quality, or code less frames at a higher spatial quality. Depending on the complexity of the content, either strategy can potentially lead to better overall quality. Methods to control this trade-off in an MPEG-4 object-based encoder have been described in U.S. Pat. No. 5,969,764, xe2x80x9cAdaptive video coding methodxe2x80x9d, issued on Oct. 19, 1999 to Sun et al., and in xe2x80x9cMPEG-4 rate control for multiple video objects,xe2x80x9d IEEE Trans. on Circuits and Systems for Video Technology, February 1999, by Vetro et al. There, two modes of operation were introduced, HighMode and LowMode. Depending on a current mode of operation, which was determined by the outgoing temporal resolution, adjustments in the way bits were allocated were made.
Besides the work referenced above, methods to control this spatio-temporal trade-off have received minimal attention. Furthermore, the information that is available in the transcoder to make such decisions is quite different than that of the encoder. As a result, the transcoder must find some alternate means of transmitting the information that is contained in a bitstream to adapt to reductions in available bit rates.
It is an object of the invention to reduce the bit rate of a video bitstream without severely degrading the information that is conveyed at the reduced bit rate. Furthermore, it is an object to reduce the bit rate by individually manipulating video objects of the bitstream in a joint manner. Because this object-based scheme has this finer level of access, it enables a reduction of the level of spatio-temporal quality of one object without significantly impacting the quality of the entire bitstream.
More particularly, the invention provides a bitstream scaler and method that operates individual object of a compressed bitstream. A demultiplexer extracts each video object from a compressed input bitstream as an elementary input bitstream having a first bit rate. A transcoder converts each elementary input bitstream to an elementary output bitstream having a second bit rate. A transcoding control unit, coupled to the transcoder, generates control information for the transcoder. A multiplexer composes the elementary output bitstreams into a compressed output bitstream having the second bit rate. Typically, the second bit rate is less than the first bit rate. The scaler also includes an output buffer generating rate feedback information for the transcoding control unit. The trancoding control unit can include a shape analyzer, a texture analyzer, a temporal analyzer, and a spatial analyzer to generate the control information from object data generated by the transcoder.